Validation of the Instructional Materials Motivation Survey (IMMS) in a self-directed instructional setting aimed at working with technology

The ARCS Model of Motivational Design has been used myriad times to design motivational instructions that focus on attention, relevance, confidence and satisfaction in order to motivate students. The Instructional Materials Motivation Survey (IMMS) is a 36-item situational measure of people’s reactions to instructional materials in the light of the ARCS model. Although the IMMS has been used often, both as a pretest and a posttest tool serving as either a motivational needs assessment prior to instruction or a measure of people’s reactions to instructional materials afterward, the IMMS so far has not been validated extensively, taking statistical and theoretical aspects of the survey into account. This paper describes such an extensive validation study, for which the IMMS was used in a self-directed instructional setting aimed at working with technology (a cellular telephone). Results of structural equation modeling show that the IMMS can be reduced to 12 items. This Reduced Instructional Materials Motivation Survey IMMS (RIMMS) is preferred over the original IMMS. The parsimonious RIMMS measures the four constructs attention, relevance, confidence and satisfaction of the ARCS model well, and reflects its conditional nature. Introduction In the field of educational science, the ARCS Model of Motivational Design (Keller, 1983, 1987a, b, c, 1999, 2010; Keller & Kopp, 1987) has been used myriad times to apply motivational strategies to instructional materials, and to test their effects. Although the model was originally designed to influence student motivation in a classic learning setting, with face-to-face interaction between teacher and students, by now it has also been thoroughly applied to and tested in other settings like computer-assisted instruction, and computer-based and distance education (eg, Astleitner & Hufnagl, 2003; Bellon & Oates, 2002; Chang & Lehman, 2002; Chyung, Winiecki & Fenner, 1999; Keller, 1999; Shellnut, Knowlton & Savage, 1999; Song & Keller, 2001). And in recent years, the ARCS model has been applied to and tested in self-directed, print-based instructional settings, applying it to cell phone user instructions and testing for effects British Journal of Educational Technology Vol 46 No 1 2015 204–218 doi:10.1111/bjet.12138 © 2014 British Educational Research Association on users likely to benefit from motivational instructions (see Loorbach, Karreman & Steehouder, 2007, 2013, b for elaborate descriptions). Keller’s publications on the ARCS model show a similar expansion of its scope as other publications over time. In his early work, Keller (1987a, b, c) speaks of “students’ motivation to learn,” “education,” “course,” “lesson” and “classroom setting.” In 1999, he states that “it is one thing to design for learner motivation in a classroom setting where teachers or facilitators can respond to changes as soon as they sense them. It is a greater challenge to make self-directed learning environments responsive to the motivational requirements of learners” (p. 39). The ARCS Model of Motivational Design The ARCS Model of Motivational Design is based on an extensive review of the motivational literature, which led to a clustering of motivational concepts into four constructs: (A)ttention, (R)elevance, (C)onfidence and (S)atisfaction (Keller, 2010, p. 44). According to Keller (2010, pp. 44–45), the following goals have to be met for people to be motivated to learn: (A) People’s curiosities and interests should be stimulated and sustained. (R) Before people can be motivated to learn, they will have to believe that the instruction is related to important personal goals or motives and feel connected to the setting. (C) Even if people believe the content is relevant and they are curious to learn it, they still might not be appropriately motivated due to too little or too much confidence, or expectancy for Practitioner Notes What is already known about this topic • The ARCS Model of Motivational Design has been used myriad times to design motivational instructions in a wide range of educational settings (from traditional to computer-assisted instruction and distance education). • The Instructional Materials Motivation Survey (IMMS) has been used often to measure people’s reactions to (motivational) instructions. • Several researchers have attempted to validate the IMMS before. What this paper adds • This paper describes an extensive validation of the IMMS. • The IMMS is validated using the results of studies that applied the ARCS model and the IMMS to motivational instructions in a self-directed instructional setting. Participants (seniors between 60 and 70) were likely to benefit from motivational instructions and used the instructions in a self-directed instructional setting. • This validation results in a reduced version of the IMMS that consists of 12 items: the RIMMS. Implications for practice and/or policy • In self-directed instructional settings with users likely to benefit from motivational instructions, the parsimonious Reduced Instructional Materials Motivation Survey (RIMMS) is preferred over the original IMMS to measure people’s reactions to motivational instructions. • The RIMMS measures the four constructs attention, relevance, confidence and satisfaction of the ARCS model well. • The RIMMS also reflects the conditional nature of the underlying ARCS model. Validation of the IMMS 205 © 2014 British Educational Research Association success. They could have well-established fears of the topic, skill, or situation that prevent them from learning effectively. Or, at the other extreme, they might believe incorrectly that they already know it and overlook important details in the learning activities. Keller (2010, p. 46) states that being successful in achieving these first three motivational goals (attention, relevance and confidence) results in people being motivated to learn. (S) To have a continuing desire to learn, people must have feelings of satisfaction with the process or results of the learning experience. The above description of the ARCS model is visualized in Figure 1. Practical strategies described in the ARCS theory and in the Motivational Tactics Checklist (see Keller, 2010, pp. 287–291) can be used to achieve each of the four goals. To measure whether these goals have been met and to measure learners’ motivational needs prior to applying ARCS strategies, Keller (2010, pp. 277–286) designed the Instructional Materials Motivation Survey (IMMS), a 36-item situational measure of people’s reactions to instructional materials in the light of the ARCS model. As such, it measures people’s scores on an attention, relevance, confidence and satisfaction construct, cumulatively resulting in an overall motivation score. Effects of motivational instructions in a self-directed instructional setting A previous study on the effects of ARCS-based motivational instructions in a self-directed instructional setting tested for effects of three motivational manipulations in cell phone user instructions respectively, focusing on attention, relevance and confidence (see Loorbach et al, 2007 for an elaborate description). Seventy-nine Dutch senior participants between 60 and 70 years of age filled out questionnaires and performed three tasks with a cell phone, using either a control version or one of three motivational versions of the user instructions. Participants in this study were seniors because they belong to a user group that is known for being less experienced with relatively new technology devices like cellular telephones (Schwender & Köhler, 2006) and are therefore more likely to benefit from motivational instructions. Results showed that participants using either a version of the instructions that focused on relevance or a version that focused on confidence performed more tasks correctly than participants in the control condition, using instructions without motivational manipulations. This study also showed positive effects of motivational instructions on behavior-deduced motivation. For this measure, we only included participants who did not complete the task, and we checked whether they felt too frustrated and gave up prematurely, or persisted and their efforts were stopped by the researcher after they had been working on the task for 15 minutes. Results showed that participants using the confidence-focused user instructions persisted in working on the third task, where they had to edit a contact’s phone number, more often (p < .05). A tendency toward a similar effect existed for the first task, where they had to change the cell phone’s ring tone (p < .10). So even though the ARCS Model of Motivational Design was not originally designed to increase user motivation in self-directed instructional settings, its potential was discovered for such settings. Its potential was especially discovered concerning confidence-focused instructions, which Figure 1: ARCS model of motivational design 206 British Journal of Educational Technology Vol 46 No 1 2015 © 2014 British Educational Research Association positively affected participants’ task performance and their persistence in trying to complete tasks. This is in line with the expectations of the ARCS model: when it was first developed, Keller (1987c) stated that “differences in confidence, the third major component of the model, can influence a student’s persistence and accomplishment” (p. 5). However, even though the behavior of participants using the control version and participants using the confidence version of the instructions statistically differed in persistence, these findings were non-existent according to their motivation scores on the IMMS. A possible explanation is that participants who used the motivational instructions did have an increased motivation level but were not aware of it, and therefore a self-report measure like the IMMS did not pick up on it, even though their behavior showed otherwise. According to Song and Keller (2001), “the use of self-report methods for measuring motivation [is] limited in that such methods [require] students to indicate their perceived motivation level, which might have been different from their actual amount of effort—a more accurate measure of motivational behavior” (p. 20). The latter is a confirmation of Keller’s (1983) words: “Effort refers to whether the individual is engaged in actions aimed at accomplishing the task. Thus, effort is a direct indicator of motivation” (p. 391). Another explanation for the discrepancy between outcomes of behavior-deduced and selfreported measures of motivation is that the IMMS might not be suitable for measuring motivational differences in our self-directed instructional setting after all. The ARCS model—which the IMMS is based on—did prove effective in our setting (see Loorbach et al, 2007, 2013a, b), in spite of its original aim to increase motivation in instructor-facilitated instructional settings (Keller, 2010, p. 310). This paper describes a validation of the IMMS to rule out or confirm this second explanation: that the IMMS as it is might not be suitable for measuring motivational differences in a self-directed instructional setting.